Systems and methods for spatial and temporal experimentation on content effectiveness

ABSTRACT

Systems and methods for organizing and controlling the display of content, then measuring the effectiveness of that content in modifying behavior, within a particular temporal and spatial dimension, so as to minimize or eliminate confounding effects.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/021,991, filed Jun. 28, 2018, allowed, which is a continuation ofU.S. application Ser. No. 15/114,440, filed Jul. 27, 2016, now grantedas U.S. patent Ser. No. 10/142,685, which is a national stage filingunder 35 U.S.C. 371 of PCT/US2015/013167, filed Jan. 28, 2015, whichclaims the benefit of U.S. Provisional Application No. 61/933,029, filedJan. 29, 2014, the disclosures of which are incorporated by reference intheir entireties herein.

BACKGROUND

Internet commerce grew massively and continues to grow, in part becauseit is a medium for marketing and communication that is susceptible toexperimentation, with a well-defined feedback loop established by theact of clicking on a piece of content to follow a link. The simplicityof such a relationship between the content and the response, along withmeans of defining who is acting, that enables the testing andoptimization of content through means such as Google Analytics, and hasenabled business models such as pay-per-click due to the naturallyclosed loop of internet behavior.

A variety of communications channels have emerged in the last few years.They include social media, mobile devices, IPTV, in-store digitalsignage and digital billboards, among others. These channels arecharacterized by a great deal of control over the content that ispresented and the ability to change the content readily. Most of thesecannot be directly linked to many important viewer behaviors such aspurchasing decisions, because they lack a distinct interactive behavior(analogous to a “click” in internet advertising). There is stronginterest in finding ways to identify a return signal for each of thesemeans of content presentation, both in forms that can isolate individualchannels and for integrated systems across those channels that canmeasure the effects of combined content received from those multiplechannels, and for purposes ranging from improving the effects ofadvertising campaigns to enhancing public health messaging to improvingtraffic control technologies.

Current efforts to capture the impact of content on behavior arecentered either on token creation or data mining. Token creationinvolves introducing some additional behavior to link the promotion toan individual's purchase through methods such as Microsoft TAG,couponing programs such as Groupon, loyalty rewards programs andcheck-ins such as social media platforms. These approaches tend toproduce small and biased samples as a result of the need to opt-in ortake additional steps to utilize the token, and typically suffer fromincreased cost and complexity due to a need to actively induce users toparticipate in the token system. They also struggle to be adoptedbecause the additional required behavior enabling measurementnecessarily alters the within-location experience. Data mining requiresa significant volume of data, and is limited to correlation studies, notactive cause-and-effect experimentation.

SUMMARY OF THE INVENTION

The present invention is directed to systems and methods for managingdelivery of content to a network of digital content presentationdevices, to implement an experimental design allowing the content servedto be associated with changes to viewer behavior for a period of timeand a given area of space.

The present invention comprises generating spatial-temporal experimentalunits based on content channel factors, assigning the spatial-temporalexperimental units to a hierarchical structure, assigning content to thespatial-temporal experimental units based on an experimental design andthe hierarchy of such units, and collecting data on the effectiveness ofcontent assigned to the hierarchy of spatial-temporal experimentalunits.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart outlining the process of getting feedbackregarding content distributed across multiple digital channels.

FIG. 2 is a flowchart outlining the creation of hierarchicalspatial-temporal units.

FIG. 3 is one embodiment of a map displaying the format of a hierarchyof spatial-temporal units.

FIG. 4 is a flowchart outlining the assignment of content to anindividual channel for measurement of content impact considering justthat channel.

FIG. 5 a is a flowchart outlining the assignment of content to multiplechannels for measurement of cross-channel impacts regarding contenteffectiveness.

FIG. 5 b is a flowchart outlining the assignment of content to multiplechannels for measurement of cross-channel impacts regarding contenteffectiveness.

FIG. 5 c is a flowchart outlining the assignment of content to multiplechannels for measurement of cross-channel impacts regarding contenteffectiveness.

FIG. 6 is a system diagram outlining systems to implement the process ofgetting feedback regarding content distributed across multiple digitalchannels.

FIG. 7 is a system diagram outlining the types of data shared among themodules of a system configured to implement the process of obtainingfeedback regarding content distributed across multiple digital channels.

DETAILED DESCRIPTION

The off-line world demands analogous capabilities to test theeffectiveness of content in influencing behavior and in selectingcontent to drive particular behaviors and outcomes. However, theoff-line world does not readily allow for the kinds of unconfoundedcontent presentation that websites do, and lack the closed feedback loopof the internet, since the content and the desired behavior lack theclosed loop made possible by the internet, and where more confounds canmuddy the relationship between the content and behavior.

The closed loop fails to exist in the off-line world because therelationship between off-line communications content and off-linebehaviors is far more difficult to track and far more complex. Insteadof being served content in response to a discrete request for a website,and interacting directly with the content through a click, content comesthrough multiple channels and is displayed regularly or continuouslywithout user input, and the desired behavior is usually undertakenthrough means that do not require interaction with the content, such aspurchasing a particular item, or moving from one particular regionwithin a location to another. The physical world also has more layers,or channels, of media presentation than the online economy. For example,and as further discussed below, channels include digital in-storeadvertising; content presented on a user's mobile phone; contentprovided to a user on his or her computer; content displayed on aroadside digital sign, etc. As a result, it is essential for measurementand optimization techniques to be able to develop understandings ofinteractions among the various channels of media presentation, andcoordinate the content presented on such multiple channels.

Conducting experimental research generally requires an experimentalunit, and the development and implementation of an experimental design,including the management of confounds. It has been discovered that toeliminate potential confounds and structure an experiment for a channelof content communication, spatial and temporal reach factors of aparticular channel are accounted for in defining experimental units,assigning content to those experimental units, and collecting andprocessing data from those experimental units.

The temporal reach factor, generally, is the period of time fromexposure to the channel until it is likely that a content recipient hastaken an action in response to the content. In some embodiments, thisperiod of time is essentially a confidence interval regarding the amountof time a population spends between receiving a message and acting uponit through a purchase or entering a location or other such behaviorsresponsive to the content, such as the time between observing a digitalsign and making a purchase at a location, or the time between viewingcontent delivered on a mobile device and visiting a location promoted bythat content. Temporal reach factors may be estimated by system usersand content presenters, or may be derived from data such as observationsof consumer dwell time, marketing studies, and other data reflectingactual content recipient behaviors relevant to the time during which thecontent recipient may be influenced by the content. The temporal reachfactors may be represented as a likelihood that the behavior may beobserved within a given amount of time, for example the period of timewhich is the 95% confidence interval that someone entering a store hascompleted their final transaction for that visit may be used as thecustomer visit duration, the temporal factor for in-store digitaldisplays.

The spatial reach factor, generally, is the region in which a contentrecipient will likely take action after receiving content. This is oftenbased on the particular channel used, typically due to the desiredimpact of the message given the content. For example, content deliveredby digital signs in a store environment is typically directed toaffecting the visit to that store environment, while content deliveredto the mobile device of a user outside of a store environment istypically directed to pushing a recipient to travel to a nearby locationand undertake an activity, such as purchasing a searched-for item, and adigital billboard presents content directed to suggesting an activity toundertake during trips passing by that billboard. The expected regioncan be based on raw distances (such as a certain radius around the pointat which content is delivered) from the point of content delivery, orderived from data such as traffic data, consumer behavior data linkingregions to particular stores, demographic and socioeconomic dataregarding regions of content delivery and responses to content by thoseregions, market segmenting data for geographic regions, or other datathat is indicative of the likely regions in which content viewers orrecipients will act in response to that content. This may also be basedon probabilities of action, such as the 95% confidence interval for thearea within which a viewer of a hand-washing reminder displayed in ahospital would use a hand sanitizer dispenser may be used as the spatialreach factor for that hand-washing reminder.

Channels for content delivery, generally, are digital displays, in someembodiments having common control, and sharing characteristics such asthe general category of display, the locations where displays on thechannel are situated, and the purpose of the displays. The displays in achannel share temporal and spatial factors due to their common generalnature and purposes, and in embodiments of this method, are subject tocommon control of the pieces of content that are displayed. This controlof the pieces of content displayed may be exercised through specificselection of content for display, providing playlists to the displays,providing rules for percentages of content play, providing contentelements for assembly into content and/or providing rules governing theautomatic generation of content to the players. The displays within aparticular channel for content delivery may differ in various ways, suchas the specific model or dimensions of the digital display. The channelmay, in some embodiments, be associated with an intermediate metric,which is used to identify the particular effectiveness of that channelfor content distribution through measuring behaviors driven by thatchannel for content delivery.

One example of a channel for content delivery is fixed-location digitaldisplays within a subsection of a location such as a particular sectionof a store or one intersection in a downtown area. Content in theseapplications may be, for example, content directing viewers to visitother subsections of the same store for related items, or presentingroute information specific to that one intersection. Within a subsectionof an area, the temporal reach factor may be the quantity of time avisitor spends in that particular subsection and the spatial reachfactor may be the boundaries of that particular subsection. Forfixed-location digital displays managed on a subsection level, theintermediate metric may be, for example, traffic in adjacent subsectionsof the store.

In some embodiments, a set of fixed digital displays for an entirelocation such as a store or a hospital may constitute a single channelfor content delivery. In these embodiments, the temporal reach factormay, for a store, be the customer visit duration, the amount of time aconsumer spends from entering the store location until the last time atwhich at least a significant percentage (usually 95%) of customers willhave completed any transactions they would make in the store location.The spatial reach factor is a radius within which the content displayedin the location will impact behavior; this is based on behavior datacollected about visitors and may extend beyond the location. For fixeddigital displays affecting a whole store location, the intermediatemetric may be, for example, rates of interaction with an in-store kiosk.

In some embodiments, the channel of content delivery may be providingcontent to mobile devices that are available to receive content. Inthese embodiments, the temporal reach factor is a recipient responseperiod, the amount of time from receiving mobile content until the endof the period most content recipients (typically 95%) have acted or notacted in response to received content. The spatial reach factor is now aregion around the point of content delivery. In some embodiments, theregion is generated based on the specific point of content delivery. Inother embodiments, the region is based on the point of content deliverybeing within a sub-region. The recipient response period and the regionmay be defined based on a particular radius from the point of contentdelivery, or may be defined using more complex models of contentrecipient behaviors, based on data such as socioeconomic data, trafficflow maps, consumer profile data, store data, rewards programs, andother data indicative of where a content recipient would act on receivedcontent. The location of the mobile device may be estimated throughvarious known location services such as IP address, location services,GPS, cell tower triangulation or other means by which the devicecommunicates its location in physical space. An example of anintermediate metric for mobile device content is interactions withreceived pieces of mobile content, such as redemptions of offersdistributed to mobile devices.

Other channels for content delivery include IPTV, web content requests,or social media activities. Each of these has temporal reach factorsbased on a recipient response period, defined as above and calculatedseparately for each particular channel of content delivery and observedcontent recipient behavior from that channel. The spatial reach factorfor each of these channels is a region around the point of contentdelivery, generated either on the specific point of delivery, or basedon delivery of content within a specific sub region. The region may bebased on a distance around the point of delivery, typically a distanceselected based on consumer behavior patterns, or defined morespecifically through models of content recipient behaviors, based ondata such as socioeconomic data, traffic flow maps, consumer profiledata, store data, rewards programs, and other data indicative of where acontent recipient would act on received content. The location of thedevice receiving and presenting content may be estimated through variousknown location services such as IP address, location services, GPS, celltower or wireless signal triangulation or other means by which thedevice communicates its location in physical space. An example of anintermediate metric for internet-based activities such as social media,IPTV and internet content may, for example, be click-through rates onads placed in these contexts.

Another channel for content delivery is digital billboards. For digitalbillboards, the temporal reach factor is a viewer response duration. Insome embodiments, the spatial reach factor is a region surrounding onebillboard; in other embodiments, it is a region surrounding a number ofnetworked billboards. The region may be defined in some embodiments by adistance around the billboard or billboards, typically a distanceselected based on consumer behavior patterns, or defined morespecifically for the billboard or billboards through models of contentrecipient behaviors, based on data such as socioeconomic data, trafficflow maps, consumer profile data, store data, rewards programs, andother data indicative of where viewers of the billboard or billboardswould act on the presented content. An example of an intermediate metricfor mobile device content is the number of visitors to a store locationwithin the response area.

Other channels of content delivery may also be used in some embodimentsof the present invention, accounting for the particular spatial andtemporal reach factors associated with those channels based on thegeneral nature of the temporal reach factor as the time between contentexposure and the probable occurrence of related behavior, and thespatial reach factor as the region over which behaviors are likely tooccur in response to content presented on a given channel for contentdelivery. Optionally, those channels of content delivery may beassociated with intermediate metrics used to measure the particularimpact of those channels.

FIG. 1 is a flow chart of a method for conducting an experiment oncontent effectiveness across a plurality of channels for contentdelivery. In some embodiments, the method includes the generation ofHierarchical Temporal-Spatial Units (HTSUs) 102. The channels forcontent delivery involved in the experiment are determined, and HTSUsare defined for each channel of content delivery included in theexperiment, based on the spatial and temporal reach factors particularto each of those channels. The channels involved in the experiment maybe identified by user input, or by querying all content delivery devicesavailable to a system implementing this method. For each channel forcontent delivery, the scope of the HTSU is defined according to thetemporal and spatial reach factors of that channel for content delivery.Temporal reach factors are the period over which content delivered by achannel may influence behavior; they may be expressed as a probabilityof an action (or the corresponding failure to act) occurring within atime window following the content exposure. The temporal reach factor isparticular to the channel being used to deliver or present content, andmay also be influenced by the type of content being presented, such asadvertisements for different products or services that are purchased ondifferent schedules or with differing levels of consumer deliberation,or the transient nature of some behaviors such as following a series ofdirections to an event in a location. Temporal reach factor data may bederived from such data as traffic flow observations, store transactionrates, and observed customer visit durations, or other statisticalmeasures of the period between content exposure and action potentiallyinfluenced by the content; this includes population statistics andconfidence intervals observed regarding the duration of between exposureand action across populations. Content exposure may, depending on thechannel of communication, be determined based on being in proximity tothe presentation of the content, or the delivery of that content to adevice such as a mobile device or computer accessing a webpage. Theaction may be unrelated to the means of content delivery, such as makinga purchase at a store check-out register after seeing an advertisementon a nearby digital billboard. The HTSU has a response duration based onthis temporal reach factor to ensure that there exists a period of eachHTSU where the actors undertaking the observed behaviors could have beenexposed to the content of the current HTSU, but not of the content inthe prior HTSU for the same content channel. In some embodiments, theresponse duration is calculated by receiving the temporal reach factoras a period of time, such as the period for a store during which 95% ofvisitors to the store have completed their final transaction of thatvisit, and then doubling that period of time to produce the responseduration for the HTSUs of that channel.

The spatial reach factors are based on the area over which contentpresented via a display on a particular channel for content delivery mayinfluence behavior. The spatial reach factor may be based upon data suchas traffic data, consumer profile data linking regions to particularstore locations, demographic and socioeconomic data regarding regions ofcontent delivery and responses to content by those regions, marketsegmenting data for geographic regions, or other data that is indicativeof the regions in which content viewers or recipients will likely act inresponse to that content with a known or estimated probability. The HTSUhas a response area based on this spatial reach factor to ensure thatdata recorded within the response area is unlikely to have beeninfluenced by content delivered on the same channel in adjacent regions.For example, the response area of the HTSU may be derived by receivingthe temporal reach factor as a radius within which viewers of contentare likely to act, such as the 95% confidence interval for the distancepeople will travel to act on a piece of content they received on theirmobile device, and doubling that radius to find the radius for theresponse area of the HTSU.

Once the scope of the HTSUs has been defined for each channel, thedisplay locations and times those displays are functioning are parsedinto specific HTSUs having response areas of the determined size andresponse durations of the determined duration. The HTSUs may be orderedinto hierarchies based on the scope of each different channel's HTSUsdetermined in step 102. This process of building hierarchies of HTSUs isdetailed in FIG. 2 , which represents an embodiment of the process forgenerating a hierarchy of spatial-temporal experimental units which canimplement a multi-channel experiment on content effectiveness. HTSUs aregenerated for the available channels 202, based on the spatial andtemporal reach factors for each channel, as described in step 102 ofFIG. 1 .

The channel having the largest HTSU is selected 204 based on the size ofthe response area of the HTSUs of the different channels. Once thechannel having the largest HTSU is selected 204, a content servinglocation on that channel is randomly selected 206. The selection israndom on the first iteration, and on subsequent iterations, therandomness is constrained such that the selection is randomized amongall content serving locations that are currently not yoked to anidentified group. A group is created by yoking all content servinglocations within the response area of the selected content regiontogether 208, and assigning a unique identifier to that group 208.Yoking together content serving locations within an HTSU associatesthose locations and ensures that they are associated with one anotherand will be assigned the same content during the response duration ofthe HTSU, controlling the content delivery to ensure that contentserving locations do not confound an experimental design and thatbehaviors observed in the response area can be tied to specificpresentations of content. The response area of the content is based onat least twice the size of the spatial reach factor for that particularchannel. This process is iterated until all content serving locationshave been assigned to a group, by checking for unyoked content servinglocations 210.

Once it has been determined that all content serving locations for thelargest HTSU have been yoked into groups 210, the channel having thenext largest HTSU is selected 212 or the HTSUs for that channel aregenerated in accordance with 102. For this next-largest channel, againdetermined by the spatial reach factor for the channel, a group createdin 208 or 218 for the channel above the current channel is selected atrandom 214. A content serving location for the channel and within theselected group is selected at random 216 from among the locations notcurrently yoked to a group. The randomization of the selection in 216 isconstrained to prevent a content serving location from being selected ifit is already yoked into a group. The selected content serving locationand all content serving locations using that channel and within theresponse area are yoked together and assigned a group identifier 218.Once a yoked group is created 218, unyoked content serving locationswithin the larger group are identified 220 and, if any are found, theselection stage 216 and yoking stage 218 are repeated until all contentserving locations within the group of the next larger channel are yoked.Once all content serving locations within a group of the next largerchannel have been yoked, another group within the larger channel isselected 214 and content serving locations are selected 216 and yoked218 if there are other groups in that larger channel that have not hadcontent serving locations within them selected 216 and yoked 218. If itis determined that all locations of a level of the hierarchy have beenassigned to groups, the channel having the next largest HTSU is selectedand it is run through this process, continuing until all content servinglocations on all channels are yoked.

Returning to FIG. 1 now, with the HTSUs themselves defined, generated,and ordered into a hierarchy 102, the attributes for each HTSU in thehierarchy are indexed 104. Attributes that are indexed for the sake ofcoordinating content presentation across the network include: the timeat which the particular HTSU begins and ends, any groups to which theHTSU is yoked, the location of content delivery for the HTSU, the regioncovered by the HTSU, and the relationships of each HTSU to other HTSUsfor different channels that are above or below the HTSU in the definedhierarchy formed by the yoking operations outlined in FIG. 2 . Thisindexed data may be stored in memory for use in subsequent aspects ofcontent assignment.

The attributes of the content pool are indexed 106. This indexing isdone for each piece in the content pool, recording the contentattributes relevant to coordinating the display of content, such as thebehavior the content is intended to influence, the displaycharacteristics of the content, identifiers or labels for the content,strategic or tactical aspects of the content regarding influencingviewer behavior, associations of the content with particularmanufacturers or business entities, and the content elements andattributes defining the content, and confound relationships betweenpieces of content. Confound relationships may be based on the observedor expected effects of the content, by other content characteristics, ordefined by specific user requests for pieces of content to be comparedto one another. This indexing may be done using, for example, automatedanalysis of the content files themselves, tags associated with contentpieces (the tagging performed either by users, or automatically based onobserved or predicted content effects, such as the behavior levelsassociated with playing a given piece of content) such as contentmetadata, content information stored in a separate database referencingthe content pieces, the specific content elements comprising thecontent, the relationships among content elements that define thearrangement of the content, or metadata applied to the content.

Optimization goals and weights are received and/or updated 108. Theseoptimization goals and weights are parameters used to determine theutility of playing certain pieces of content, either to conduct anexperiment or to optimize the expected results for the selection ofcontent to be played. Goals are metrics based on the collected dataindicative of content effectiveness and are key parameters fordetermining the utility of content, while weights are a factor used toallow for the comparison of diverse goals, such as increasing sales oftwo items that are typically in at least partial competition with oneanother, such as salads and chicken sandwiches in a restaurant. Thegoals and weights may be selected by the user to define permissibletradeoffs, or derived from business data such as the gross profitmargins on the sales of particular items. The optimization goals andweights include both within-channel parameters and cross-channelparameters. Within-channel goals include intermediate metrics measuredfor each channel, such as in-store traffic driven by digital billboardmessaging. Cross-channel goals are overall goals for a multi-channelcontent presentation pattern, such as overall retail sales or grossprofits during a given HTSU. Weighting factors may be used to convertthose goals into optimization values on a common scale, either concrete(e.g. dollars) or arbitrary (e.g. an optimization metric) for use incalculating the utilities of certain content selections that mayinfluence a set of potentially diverse goals, such as driving sales ofpartially competing items, for example, salads and grilled chickensandwiches at a quick-serve restaurant, or store-brand and name-brandvariations of a particular product in a large retailer.

Content presentation constraints are received 110. The contentpresentation constraints may include display constraints, such as thesize, orientation, resolution, and capabilities of a particular displayon a channel, and in some embodiments business constraints such asparticular content combinations that are impermissible, or contentattributes that must be set based on business constraints (for example,using highly specific color selections in a particular colaadvertisement, based on that cola's branding requirements). The displayconstraints may be received from memory where this information isstored, or received from querying the players regarding the displayconstraints. Confound relationships indexed in step 106 for the piecesof content may be treated as content presentation constraints and mayalso be part of the information received in this step, either from theindex of content attributes, or from other sources of confoundinformation, such as the nature of a particular experimental design (twocontent pieces that are being tested against one another are confoundsto one another) or the observed or predicted effects of content.

Explore and exploit utility functions are used to calculate thewithin-channel utility of playing pieces of content 112. Explore utilityfunctions determine the value of gaining additional informationregarding content effectiveness in certain channels, times, and/orlocations and the predicted impact of playing a piece of content andrecording data on improving that content effectiveness information basedon experimental power analysis, knowledge of the rate of change ofcontent effectiveness data, and data on the effectiveness and certaintyregarding the effectiveness of other pieces of content. The exploreutility function may be probability-matching strategies based on thecurrently observed content effectiveness data, where the relativefrequency with which a piece of content appears matches the likelihood,according to the content effectiveness data and the confidence intervalsfor that data, that it is the optimal piece of content to show toachieve a selected goal. Other approaches to the “multi-armed bandit”problem may also be applied to determining explore utilities.

Exploit utility functions are used to calculate the present value ofdisplaying certain pieces of content on certain channels and in certaintimes and locations. These calculate rewards for the observed effects ofcontent, such as driving gross profit observed in a store during an HTSUwhere content is playing, or reducing the number of vehicles approachingan intersection during an HTSU where alternate route data is beingpresented. These rewards and knowledge of content effectiveness may beutilized by machine-learning systems to pursue particular outcomes usingutility functions where content effectiveness data is used to determinewhat content will maximize a selected value.

Explore and exploit utility functions may also be used to calculatecross-channel utilities for playing pieces of content 114. Cross-channelexplore utility functions capture the potential value of learning moreabout the interactions between content distribution channels andparticular combinations of content delivered to users. Cross-channelexploit utility functions determine the additional benefit of particularcombinations of content across multiple channels and the influence of aparticular piece of content on the effects likely to be seen whenintegrated into the entire presentation of content across all thechannels; these may be expressed through representations such as Markovchains, Petri nets, or multiobjective utility functions, and mayincorporate effects on channel-specific intermediate metrics. As in 112,the utility function may include weighting for historical data thatdiscounts the impact of older data on the utility determinations toemphasize more current content effectiveness data in selecting contentfor display.

The utility values calculated in 112 and 114 may be used to selectcontent for play in each of the HTSUs 116. Content is algorithmicallyselected based on the expected overall utility, when combining thewithin- and cross-channel utility functions for all the involvedchannels and any weighting factors. The algorithm may be a reinforcementlearning routine, a generic algorithm, Monte Carlo methods, and othermachine learning techniques. Factors include the expected immediaterewards 124 for displaying the content (for example, a boost in salesduring the response duration of the HTSU where the content is played, asobserved in the data from prior HTSUs), the expected rewards givenspatial-temporal decay functions representing continuing effects 126(for example, hand-washing reminders engraining the habit in hospitalpersonnel, as observed in carry-over effects observed when comparingdata over time), and the expected utility of improving the causalknowledge representations 128 (for example, the benefit expected fromreducing the uncertainty about the expected rewards for a piece ofcontent, based on power analysis of experimental trials, and/or theeffects of other content on the same rewards). The assignment of contentto play in HTSUs may also ensure that confounding pieces of content arenot assigned to the same HTSU, using indexed content data 106 andcontent presentation constraints 110 to identify the confoundrelationships, and may also use the indexed content data from 106 andestablished principles of experimental design to algorithmically selectcontent such that the content assigned to the HTSUs is randomized sothat content is played under a variety of different conditions acrossthe various HTSUs, that proper amounts of each piece of content areplayed to ensure balancing of an experiment, that the content pieces areplaced in an order that can counterbalance out the possibility of ordereffects in an experiment. Content selected for display within an HTSUfor a channel for content delivery that is constantly displaying contentis displayed on the appropriate displays within that HTSU. For channelswhere content display is based on opportunities such as a request ordistributed on a push system, opportunities for content display areidentified and the selected content is presented to that opportunity,with the opportunity being defined by the ability to present content,for example as a response to a request made to access a productmanufacturer's web page, or for another example, a mobile device thathas opted into a push system for notifications regarding nearby trafficcongestion that is associated with a location that is within theresponse area of an active HTSU. The location is known through locationservices, cell tower triangulation, GPS, or other established methods ofidentifying the location of the device where content may be presented;the content selection for that active HTSU determines the type ofcontent presented to that particular opportunity. When content isassigned to an HTSU that is yoked to other HTSUs through the steps ofFIG. 2 , all of the yoked HTSUs receive the same assignment of content.

Dependent variable data is parsed by experimental condition 118. Thedependent variable data is the collected data about activities occurringwithin the response area of HTSUs in which content has been displayedand that therefore may be a response to content. Examples of dependentvariable data include sales data, captured at point-of-sale systems,activity data such as hand-washing behavior based on soap levelmeasurements at automatic hand soap dispensers, or traffic behavior suchas measured transit times or the number of cars arriving at anintersection. The experimental condition of the data comprises factorsrelating to the data and its relation to the content, including thecontent presented on each channel affecting the time and location wherethe dependent variable data was collected, and also includes the time ofthe data collection, the time at which the HTSU during which data wascollected began, the location of data collection, and other similarfactors relating to when and where the data was collected and thecontent that was played in the HTSUs proximal to the collection of thedata. The parsing 118 sorts the data by these factors and may includethe removal of data occurring during the first temporal reach factor ofeach HTSU, since that period includes data that may have been confoundedby content in the preceding HTSU. During the parsing 118, data recordedmore than one spatial reach factor from the center of the HTSU, wherecontent displayed adjacent HTSUs may have exerted influence on theobserved behavior may also be removed from the data set. The datacollection itself may be performed only during specific time periods andregions within each HTSU to remove content that may be subject toconfounds from temporally or spatially adjacent HTSUs. By removing thesepoints, the remaining data can be associated with specific pieces and/orcombinations of content presented on the various channels and theconfounds created by temporally or spatially adjacent HTSUs removed fromthe data. This parsing may separate data by type of content presented onone channel, or may parse the data by combinations of content presentedwithin a larger HTSU for a channel and the content presented on theHTSUs for smaller channels that were within the larger HTSU. The parsing118 is done algorithmically by a processor.

The parsed data from 118 is then used to update the causal knowledgerepresentation 120. The causal knowledge representation is a database ofthe observed effects of content that is parsed in step 118 andassociated with the content that was displayed at the time it wasrecorded. Updating the causal knowledge representation 120 may alsoinclude discarding old data; some relationships between content andbehavior are dynamic (for example, marketing messages on digitaldisplays in-store at fast food restaurants tend to lose effectivenessover time); for these cases, the best results may be obtained by havinga fixed data inclusion window and discarding all data that is past acertain age. It is also possible to include the age of data as a factorin the utility functions as described steps 112 or 114 to ensure propermanagement of historical data when determining the utility of playing apiece of content. In some embodiments, updating causal knowledge 120includes identifying and recording observed cross-channel effects fromdata parsed by the content presented on multiple channels within acommon HTSU as well as single channel effects, by comparing completesets of content received from each channel, as opposed to simplycomparing observed effects among variations within one particularchannel. Because content was presented in a particular manner selectedby randomization constraints and excluding confounding content fromHTSUs 116, the data recorded and then parsed by experimental condition118 may provide true experimental data and therefore causal knowledgeregarding content effects. For example, for an HTSU for a digitalbillboard where the response duration is twice the temporal factor,transactions occurring within the response area during the second halfof an HTSU's response duration could potentially be influenced by thecontent displayed on the billboard during that HTSU, but not by otherdigital billboard content, and thus may be associated with the contentfrom that HTSU.

The updated causal knowledge of content effects 122, including thecertainty regarding those content effects may then be fed into thewithin- and cross-channel utility functions of steps 112 and 114 tocalculate updated utilities for displaying pieces of content, based uponthe most recent data collected.

FIG. 3 is an example map demonstrating the structure of a hierarchy ofHTSUs created by one example of the steps of FIG. 2 and the assignmentof content to those particular HTSUs for one point in time and oneregion of a larger network, with the content selections already made inaccordance with FIGS. 1 and 2 . For this example, there are threechannels controlled by an embodiment of the invention. The channels aredigital billboards, mobile devices, and in-store displays. The channelwith the largest spatial reach factor is for digital billboard content.Mobile device content has the next largest spatial reach factor, andin-store displays have the smallest spatial reach factor. Response areasfor these HTSUs are overlaid onto a map 300, showing how these units maybe arranged spatially. The response area of the HTSUs for digitalbillboards in the example map 300 are marked by response area 308 and310. Each of these response areas may be centered on a digitalbillboard. Within these areas, digital billboards will show contentincluding the same content, variant 1 for response area 308 and variant2 for response area 310. The response area of the HTSUs for mobilecontent are marked by response area 312 and 314; within these areas,mobile content requests will produce content selected based on theassigned content of the HTSU; in this case content variant 1 forresponse area 312 and content variant 2 for response area 314. Thein-store display HTSU response areas are represented by response areas316 and 318, where the points represent the response area of the HTSUscreated for the in-store displays, the store location. Storesrepresented by store icon 316 are assigned present content variant 1,while stores represented by store icon 318 present content variant 2.The largest HTSUs contain different overall content presentations to theusers within them across all channels of content delivery. The HTSUs onthe map in response area 302, defined by a digital billboard HTSU, showsvariant 1 of the billboard content, variant 1 of the mobile content torequesters, and variant 1 of the in-store content. The response area 306shows an area where digital billboards show content variant 2, while themobile devices respond to requests with content variant 2, while thein-store displays show variant 1. Data may be compared for a givenchannel, for example comparing data from just the digital billboardHTSUs of 302 and 304 as part of a larger trial comparing the mobilecontent variants, or may be compared for all the channels of eachhierarchy, such as comparing data from all channels of content deliveryin of 302 and 306 as part of a larger trial comparing the effectivenessof particular combinations of content variants on each of the channels.Lower levels of the hierarchy do not necessarily need to all be assignedthe same content within an HTSU of a higher level of the hierarchy; thisis represented in the mobile and in-store content variants shown in area306. The HTSUs are nested so that the area covered by the response areasof higher level HTSUs completely contain the areas covered by theresponse areas of the lower-level HTSUs below them in the hierarchy, sothat observed behaviors may have been influenced by content receivedfrom all of the channels in that segment of the hierarchy.

FIG. 4 is a flowchart detailing the assignment of content to HTSUs toenable a measurement of content effectiveness free of confound effectsin some embodiments of the invention. For the purposes of assignment toan HTSU, content may include recipes or instructions for proceduralgeneration of content, rules constraining content creation, renderedfiles, percentages of play for content pieces or other informationdictating what will appear on the display. Randomization constraints arereceived in step 400; these constraints guide selection of content toallow implementation of an experimental design across a multitude ofHTSUs. Randomization constraints may be expressed as chances ofselecting a piece of content for display in an HTSU. The randomizationconstraints are generated based on explore utilities and principles ofexperimental design such as balancing. Randomization constraints, byaffecting the frequency with which content is presented in HTSUs, canbalance the experiment by ensuring content is selected at appropriaterates for a balanced experimental design; the balancing is particular tothe design for the experiment, with examples including Latin squares,partial Latin squares, randomized block designs and other designs forclinical trials. The randomization constraints may include conditionalstatements that adjust the assignments based on content selections madefor adjacent or nearby HTSUs, to allow the use of balancing andcounterbalancing to control for order effects within an experimentaldesign. An initial piece of content is assigned to a HTSU 402, selectingthe content from the available pieces of content based on therandomization constraints. Across multiple HTSUs, the assignment ofcontent in accordance with the randomization constraints may implementblocking, balancing and counterbalancing according to an experimentaldesign; this information is received in the randomization constraints400. For HTSUs that are directed towards constantly-on displays, forexample, digital billboards, the assignment of content to the HTSUdictates what will appear on screen for the duration of the HTSU. ForHTSUs that are directed towards displays that request contentperiodically, for example mobile devices, the assignment of content tothe HTSU dictates what content would be delivered in responses tocontent requests that occur during the response area and duration of theHTSU. Once the initial piece of content is assigned 402, forconstantly-on displays, the remainder of the HTSU may need to beassigned content. Content assignments to individual units must ensurethat the content within a unit is not confounded by other pieces ofcontent that are part of a trial comparing that content to a selectedpiece. Confound information for the content is received 404, definingwhat content may be played together during an HTSU and what content mustbe kept separate in order to ensure that data collected from an HTSU isnot confounded and can be used to determine content effectiveness.Confound information may be user input or may be derived from factorssuch as observed or expected content effects (content pieces affectingthe same metric may confound one another), or other content informationsuch as underlying strategy, use of particular content elements, or ofthe system testing the effectiveness of the pieces of content againstone another. The confound data received in 404 is used to ensure thatconfounding content is excluded from the HTSU while the remainder of theHTSU is algorithmically populated 406. The content is then displayed 408on the displays controlled by the HTSU, such as in-store displays, ormobile displays making a relevant request for content within theresponse area of an HTSU. During display of the content, data iscollected regarding activities occurring within the response area of theHTSU 410. This data may include such things as point-of-sale transactiondata, traffic volume data, soap consumption data, or other suchmeasurable factors indicative of behavior potentially occurring withinthe response area of the HTSU.

FIGS. 5A, 5B and 5C detail various exemplary embodiments of theinvention where content is assigned to HTSUs across multiple channels toconduct a multi-level experiment. The assignment of content may proceedin a bottom-up manner, as in FIG. 5A, or may proceed in a top-downmanner as in FIG. 5B, or may be done by assigning blocks of content tohierarchies of experimental units as in FIG. 5C.

FIG. 5A describes a top-down approach to content assignment. The channelhaving the largest HTSUs is selected 500, based on the response area ofthe HTSUs for each channel. The HTSUs of this channel have contentassigned to them 502 in accordance with randomization constraints, theassignment of content following the steps described in FIG. 4 . Once theHTSUs of the largest channel have been assigned content 502, thecross-channel constraints are received 504. Cross-channel constraintsare provided based on an experimental design and the current set ofalready-assigned content, to exclude confounds from the trials beingbuilt based on content and trial-based confound data, such as pieces ofcontent that are being compared against one another, or composed sets ofcontent that are to be presented to recipients in certain times andplaces. The cross channel constraints and the assignment of content tothe largest HTSU are used to update the randomization constraints beingused 506. This update is based on the change to the utility of exploringcross-channel effects given the state of the HTSUs that have contentassigned to them and the possibility to explore certain cross-channeleffects only for certain experimental conditions due to the assignmentof content to some HTSUs in the hierarchy. For example, when the contentassignments are made to the largest HTSUs in the hierarchy, thatdictates the combinations that are possible when controlling the contentselected for lower levels in the hierarchy. With updated randomizationconstraints, the system moves down one level in the hierarchy ofchannels 508 and uses the randomization constraints to assign content tothe HTSUs of the current channel 510. The assignment of content uses theupdated randomization constraints from 506 but otherwise follows theassignment process detailed in FIG. 4 and used in step 502. The processis repeated for each channel, until all HTSUs have been filled for thechannel having the smallest response area for its HTSUs.

FIG. 5B describes a bottom-up approach to content assignment for anexperiment including multiple channels of content communication. Thechannel whose HTSUs have the smallest response area is selected 550 andcontent is assigned to those HTSUs in accordance with the process laidout in FIG. 4 , using randomization and confound constraints to selectcontent for all the HTSUs for that channel 552. Cross-channelconstraints are received 554. The cross-channel constraints are, likethose in 504, based on a cross-channel utility function which predicts autility based on interactions observed in content effectiveness data.The system then moves to the channel with the smallest response areaHTSUs that have not been assigned content yet 556. The cross-channelconstraints and the content assigned to all prior HTSUs are used toupdate the randomization constraints 558. In a bottom-up assignment ofcontent to hierarchies of HTSUs, the randomization constraints areupdated based on the already-assigned content, which may prevent somecross-channel trials from being implemented during a particular set ofoverlapping HTSUs; the trials that may still be performed based on theexisting assignments of content at the lower levels are identified bycomparing the content assignments to the desired cross-channel trials,and the weighted randomization of assigning content to the current levelof the hierarchy is altered based on the explore values of thestill-viable trials while eliminating consideration of the explore valueof trials that may no longer be successfully implemented. The updatedrandomization constraints are used to assign content to the HTSUs of thecurrent channel of content communication 560, similarly to how contentis assigned in 502, 552 and as described in FIG. 4 . Once the currentlevel has content assigned 560, the process repeats for eachsuccessively larger (by spatial reach factor) channel for contentdelivery until the HTSUs for the largest channel for content deliver hasbeen assigned content.

FIG. 5C demonstrates a flowchart for a method for assigning content tocontent presentation opportunities, where the assignment of content forexperimental purposes uses multi-channel blocks and then fillingremaining HTSUs with exploit content. Candidate content presentationpatterns are identified for each explore trial content distributionpattern 570. Explore trial content distribution patterns are acombination of content to be delivered across multiple channels,sometimes to HTSUs having particular attributes. The content may bedefined by specific piece of content, or by the state of a number ofchannel-specific intermediate metrics. The intermediate metric for agiven channel can be driven to a particular state to implement theexplore trial content distribution pattern through content selectionthat uses existing causal knowledge of content effectiveness to predictthe state of the variable likely to result from a selection of content,and then selecting the content that is predicted to drive theintermediate metric in the proper manner. The explore trial contentdistribution patterns are generated by using the current data regardingcontent effectiveness and the confidence intervals of that contenteffectiveness data, and including cross-channel interactions to find therelative probability of one pattern of content delivery across multiplechannels being superior to another; this is computed using the exploreutility function. Content presentation opportunities are HTSUs ondifferent channels that have at least partially overlapping spatial andtemporal reach factors, and thus the content presented during each ofthe HTSUs is likely to be acted upon within the same times and places.Candidate content presentation opportunities are content presentationpatterns where the attributes of the experimental units within thecontent presentation pattern match those required by the explore trialcontent distribution pattern, with the attributes including the channelsfor content delivery present in the content presentation pattern, suchas the day part during which the HTSUs occur, the location of thoseHTSUs, details regarding those locations where the HTSUs are such associoeconomic data for the regions contained within the HTSUs, or othermarket segmenting data that may be material to conducting specifictrials of content effectiveness for particular times and locations.Candidate content presentation opportunities are detected by comparingindexed attributes for HTSUs in the content presentation opportunity andthe temporal and spatial relationships among them or the hierarchicalstructures they are organized into across channels to the particularneeds of each explore trial content distribution pattern. An exploretrial content distribution is selected randomly, without replacement572. The randomization of this selection may be completely random out ofthe set of identified explore trial content distributions, or therandomization may be weighted based on the explore utilities of thedifferent explore trial content distributions or constrained based onthe demand for particular attributes or the number of channels needed toimplement the explore trial content distribution. The selected exploretrial content distribution is then assigned randomly to a candidatepresentation opportunity 574. The set of candidate presentationopportunities is the same determined for the selected exploit trialcontent distribution pattern in step 570. The randomization of theselection of a particular candidate presentation opportunity may becompletely randomized within the set of candidate presentationopportunities or may be weighted according to the number of exploittrial content distribution patterns that a particular presentationopportunity is a candidate for, favoring the use of candidates lesslikely to be useful to other, separate exploit trial contentdistribution patterns. The assignment of a particular candidatepresentation opportunity to a particular exploit trial contentdistribution means that the HTSUs within that assigned candidatepresentation opportunity are assigned content based on the exploit trialcontent distribution pattern. This may dictate that certain pieces ofcontent appear in specific HTSUs within the candidate presentationopportunity, or that causal knowledge of within-channel effects onintermediate metrics are used to select content for each HTSU to producea given set of states for the intermediate metric for each of thechannels the exploit trial content distribution requires. Explore trialcontent presentation patterns that have un-met sample size requirementsthat can be filled are identified, and if any are found present 576, theselection and assignment stages 572 and 574 repeat until no more exploretrial content distribution patterns can or need to be assigned to theexisting content opportunities. This sample size requirement iscalculated based on the explore utility function, using power analysisand the current certainty and knowledge regarding content effectivenessto determine the number of samples for each trial to generate thegreatest impact. Content is assigned to remaining HTSUs 578 based onexploit utilities, which may be calculated based on causal knowledge ofcontent effectiveness and the confidence intervals for that knowledge.The determination of utility and assignment of content is preferablydone using a combination of a genetic algorithm and a Monte Carloprojection using the causal knowledge of content effectiveness andconfidence intervals. The content assignments may then be communicatedto the various displays that comprise the channels for content deliveryand displayed to viewers, producing a coordinated content experiencethat can be used for experimental trials, and data can be collectedregarding the behavior of people in times and locations where thesecoordinated content experiences are presented.

FIG. 6 represents an embodiment of a system of the invention. Datastorage and processors detailed in this figure may be co-located orintegrated within the same computing device. Data storage 600 include amemory storing content effectiveness data 602, a memory storing confounddata 604, a memory storing content 606, and a behavior data memory 628.These memories may be co-located or separate from one another and eachmemory may be distributed across multiple physical memory units. Thememory storing content effectiveness data contains an index of thepieces of content currently in the system (for example, in memory 606),and the current data on the effectiveness of that content in drivingbehavior, the parsed and processed behavior data from the dataprocessing unit 630. The memory storing confound data 604 contains adatabase of the relationships among content, indicating which pieces ofcontent confound each other under different experimental conditions,within and across channels of content delivery. The memory storingcontent 606 stores content files, which may include elements to be usedin content generation, rendered files, instructions for proceduralgeneration of content, rules constraining content creation, elements foruse in content creation or percentages of play for content pieces, orother information controlling what appears on a digital display.Functional modules 608 include an experimental unit creation module 610,a content assignment module 612, an experimental design module 614 and adata analysis module 630. Any of these modules may be functionallyembodied in software resident in data storage 600, then executed by amicroprocessor on a computer system. In some embodiments, the modulesmay be combined into a single module with functionality as describedherein. An experimental unit creation module 610 computes the responsearea and response duration of HTSUs based on spatial and temporal reachfactors for each channel. The experimental unit creation module 610 alsois configured to divide display time and locations into HTSUs of thedefined response area and response duration, and to organize those HTSUsinto hierarchies based on the response areas of the HTSUs for eachchannel of content communication. A content assignment module 612assigns content to HTSUs in a manner that applies randomizationconstraints from an experimental design processor 614 and confound datastored in memory 604 and in accordance with the method detailed in FIG.4 , to assign content across the HTSUs in a trial according toprinciples of experimental design including blocking, balancing,counterbalancing, and prevention of within-HTSU confounds through use ofthose randomization constraints and confound data. An experimentaldesign module 614 generates randomization constraints and may generateconfound data that is stored in memory 604. The experimental designmodule may generate confound data for memory 604 by determining whichpieces of content are within the same trial and thus confounds to oneanother. The experimental design processor applies experimental designconcepts including various types of trial types (e.g. Latin square,matched-control, etc.), power analyses for trials, and concepts such asblocking, balancing, and counterbalancing to derive the frequencies withwhich content pieces must play, the experimental conditions the contentmust play in, and constraints on content play to control for effectssuch as order effects to supply randomization constraints to the contentassignment unit 612. The content assignment unit 612 applies thoserandomization constraints and may also assign other content to presentin the HTSUs to enhance one or more effectiveness metrics based onutility functions, whose output is calculated by the content assignmentunit using content effectiveness inputs from a memory storing contenteffectiveness data 602. The content assignment unit also distributes thecontent assignments to the various content channels 616, through meanssuch as servers for mobile content, player devices for displays ondigital signage networks, player devices for digital billboards, or webservers for general delivery of web pages, social media or IPTV. Thecontent assignment unit is communicatively coupled to the variouscontent channels. The content channels 616 present the content accordingto the nature of the channel and the received assignment of content,with examples including servers for mobile websites 618 and ordinarywebsites 624, in-store digital displays 620 and digital billboards 622.For an example directed towards marketing, the in-store display channel620, players receive the content assignments and the content data andpresent the assigned content on digital displays situated in storeenvironments, while the server for a mobile website 618 for the mobiledevice display channel receives requests for content related to asubject and presents the relevant assigned content when a request ismade, and the players for the digital billboards 622 present theassigned content on digital billboard display screens. Relevantbehaviors occurring within the response area, such as purchases,navigating traffic, or using a disinfectant dispenser are recorded atone or more data recorders 626 situated in the response areas for theHTSUs, recording and storing the data in a response data memory 628. Inan exemplary embodiment directed towards marketing, the data recorders626 could be point-of-sale systems recording the transactions occurringat each data recorder. In some embodiments, the data recorders 626include both devices to capture an overall metric such as point-of-salesystems for capturing retail transactions, as well as sensors forintermediate metrics for each channel for content delivery, such asstore traffic data monitors testing the impact of digital billboards onstore traffic. The recorded response data in the response data memory628 and the content assignment information are processed by a dataprocessing unit 630 to associate the behavior data with the content thatcould have influenced the behavior. The data processing unit 630 derivesthe effectiveness of content at altering one or more effectivenessmetrics from the behavior data associated with each piece of content andcomparisons of the different effects of the pieces and combinations ofcontent and updates a memory storing content effectiveness data 602,storing the relationships observed between the content presented and thebehaviors recorded in the response area during response durations by thebehavioral data recorders 626.

FIG. 7 represents the flow of data through a system according to oneembodiment of the present invention. Behavioral data recorders 726collect behavior data and the recorded behavior data 750 is transferredto a behavior data memory 728. In some embodiments, the data recorders726 include both devices to capture an overall metric such aspoint-of-sale systems for capturing retail transactions as the behaviordata, as well as sensors for intermediate metrics for each channel forcontent delivery, such as such as store traffic data monitors testingthe impact of digital billboards on store traffic; this intermediatemetric data is included in the behavior data 750 that is transferred tothe behavior data memory 728. The data is transferred among the memoriesand processors by a variety of means, including Ethernet in local areaor wide area networks, the internet, wireless communications protocolssuch as the 802.11 standards, Bluetooth, or direct physical connectionsbetween the memories and processors. Stored behavior data 752 in thebehavior data memory 728 is transferred to the data analysis processor730. The data analysis processor parses the data by experimentalcondition (for example, the time and location of the experimental unitwhere the data was captured, or the state of intermediate metrics onother channels within the response are and response duration of theexperimental unit), producing parsed content effectiveness data 754.This parsed content effectiveness data 754 is transferred to the contenteffectiveness memory 702. From the content effectiveness data, data onthe level of certainty of content effectiveness knowledge 756 isprovided to the experiment design processor 714. The level of certaintyof content effectiveness knowledge is based on a power analysis of theamount of data and size of the observed effects. The contenteffectiveness memory also includes observations of confounding effects,where pieces of content exhibit effects on the same or relatedbehaviors, and those confound observations 768 are transferred to andstored in the confound data memory 704. The content memory containscontent 762 and the confound data associated with individual pieces ofcontent. The content confound data 770, which includes tags associatedwith content pieces (the tagging performed either by users, orautomatically based on observed or predicted content effects, such asthe behavior levels associated with playing a given piece of content),content metadata, or content information stored in a separate databasereferencing the content pieces, is transferred to and stored in theconfound data memory 704. The confound data memory 704 contains data onpotential confounds based on observed content effects 768, or fromcontent tags, metadata, strategies and user inputs 770, and thiscombined confound data 758 is provided to the content assignmentprocessor. The content assignment processor uses confound data 758 andrandomization constraints 766 received from the experiment designprocessor 714 to select content assignments 760. The content assignment760 may be a playlist, percentages or likelihoods of play within anHTSU, or other means of directing the display of particular pieces ofcontent within an HTSU. In some embodiments, content effectiveness data772 is fed into utility functions by the content assignment processor712 to select additional content that is also a part of the contentassignment 760. This content assignment 760 and the content 762 from thecontent memory 706 are supplied to the channels for content delivery 716including mobile and standard website servers 718 and 724, and playersfor in-store displays 720 and for digital billboards 722, which displaythe content 762 in accordance with the content assignments 760. Contentrecipients then may act on the content in the response areas for thesystem, with those actions recorded by behavioral data recorders 726 andthe data flowing through the system iteratively.

In addition to the embodiments claimed below, other embodimentsdescribed herein include:

A. A computer-implemented method for measuring content effectiveness,comprising:

-   -   generating a plurality of trial content distributions;    -   identifying, using a processor, candidate content presentation        opportunities on a plurality of channels for content delivery        for each of the plurality of trial content distributions;    -   selecting a trial content distribution;    -   assigning, using a processor, the selected trial content        distribution to a candidate content presentation; and    -   displaying the assigned trial content distributions during the        candidate content presentation to which it was assigned, on the        plurality of channels for content delivery.

B. The computer-implemented method of Embodiment A, wherein the trialcontent distribution comprises a required state for an intermediatemetric for each of a plurality of channels for content delivery.

C. The computer-implemented method of Embodiment B, wherein the trialcontent distribution further comprises a set of required experimentalunit attributes.

D. The computer-implemented method of Embodiment A, wherein theselecting a trial content distribution is randomized.

E. The computer-implemented method of Embodiment A, wherein thecandidate content presentation opportunities comprise a plurality ofexperimental units having overlapping spatial and temporal reachfactors.

F. The computer-implemented method of Embodiment A, wherein theidentifying candidate content presentation opportunities comprises:indexing attributes of the experimental units for each contentpresentation opportunity, and comparing the indexed attributes to a setof requirements for the trial content distribution.

G. The computer-implemented method of Embodiment A, wherein theassigning the selected trial content distribution to a candidate contentpresentation opportunity is randomized.

H. The computer-implemented method of Embodiment A, wherein theplurality of channels for content delivery comprise digital signagenetworks, mobile devices, and digital billboards.

I. The computer-implemented method of Embodiment A, wherein theassigning of the selected trial content distribution to a candidatecontent presentation opportunity comprises: receiving intermediatemetric data for each channel for content delivery; using theintermediate metric data and the trial content distribution, selectingcontent that produces the required state of the intermediate metric foreach channel for content delivery; and, assigning the selected contentto be displayed on each channel for content delivery during the selectedcontent presentation opportunity.

J. The computer-implemented method of Embodiment A, further comprisingassigning additional content to content presentation opportunities basedon an aggregate utility.

K. The computer-implemented method of Embodiment A, wherein theaggregate utility is calculated by functions comprising a geneticalgorithm and a Monte Carlo simulation.

L. The computer-implemented method of Embodiment A, further comprisingcollecting data on a content effectiveness metric.

M. The computer-implemented method of Embodiment A, further comprisingcollecting intermediate metric data for each channel for contentdelivery.

N. A computer-implemented system for measuring content effectiveness,comprising:

-   -   a software module configured to generate a trial content        distribution listing;    -   a computer memory storing candidate content presentation        opportunities available on a plurality of channels for content        delivery;    -   a content assignment software module configured to assign trial        content distributions to candidate content presentations; and,    -   a plurality of content delivery channels that receive data        describing assigned content, then present the assigned content        distributions during the candidate content presentations.

O. The computer-implemented system of Embodiment N, wherein the trialcontent distributions comprise at least one piece of content for each ofthe plurality of channels for content delivery.

P. The computer-implemented system of Embodiment N, wherein thecandidate content presentation opportunities comprise a plurality ofexperimental units having overlapping spatial and temporal reachfactors.

Q. The computer-implemented system of Embodiment N, wherein the contentassignment module assigns trial content distributions to candidatecontent presentations comprises: receiving intermediate metric data foreach channel for content delivery;

-   -   selecting content that produces the required state of the        intermediate metric for each channel for content delivery based        on the intermediate metric data and the trial content        distribution; and,    -   assigning the selected content to be displayed on the displays        of each channel for content delivery during the selected content        presentation opportunity.

R. The computer-implemented system of Embodiment N, wherein the channelsfor content delivery comprise digital signage networks and/or mobiledevices.

S. The computer-implemented system of Embodiment N, further comprisingsensors for collecting data on an effectiveness metric.

T. The computer-implemented system of Embodiment N, wherein the sensorscomprise point-of-sale systems.

U. The computer-implemented system of Embodiment N, further comprisingsensors for collecting data on intermediate metrics associated withchannels for content delivery.

V. The computer-implemented system of Embodiment N, further comprising aprocessor configured to assign additional content to unassignedcandidate content presentations based on a utility function.

W. The computer-implemented system of Embodiment V, wherein the utilityfunction is calculated using a Genetic Algorithm and a Monte Carloprojection.

These and other embodiments are described herein.

What is claimed:
 1. A computer-implemented method comprising: receivingtemporal reach data and spatial reach data for each content deliverychannel; defining at least one hierarchical temporal-spatial unit (HTSU)based on the temporal reach data and the spatial reach data, wherein theat least one HTSU has a response area based on the spatial reach data;deriving the response area of the at least one HTSU by receiving atemporal reach factor radius; modifying the temporal reach factor radiusto find a response area radius of the at least one HTSU; receivingconfound data for at least some of a plurality of different pieces ofcontent, wherein confound data includes data about which pieces ofcontent are being compared against one another; using a processor of acomputer, assigning the plurality of different pieces of content to theplurality of content delivery channels, wherein two pieces of contentbeing compared against one another are not assigned the at least oneHTSU.
 2. The computer-implemented method of claim 1, further comprisinga plurality of HTSU's.
 3. The computer-implemented method of claim 2,wherein the temporal reach factor is the same among at least two of theplurality of HTSU's.
 4. The computer-implemented method of claim 2,wherein the temporal reach factor differs between at least two of theplurality of HTSU's.
 5. The computer-implemented method of claim 2,wherein the response area is the same among at least two of theplurality of HTSU's.
 6. The computer-implemented method of claim 2,wherein the response area differs between at least two of the pluralityof HTSU's.
 7. The computer-implemented method of claim 1, furthercomprising receiving behavior data within the response area of the HTSUduring at least a portion of the response duration, wherein thecollecting of behavior data is associated with a location other than thelocation associated with the presenting of the displayed content.
 8. Thecomputer-implemented method of claim 1, further comprising: causing theselected content to be displayed on the channel for content deliveryconsistent with the content assignment; and receiving response dataindicative of the effects of the displayed content.
 9. Thecomputer-implemented method of claim 8, further comprising: determiningthe effectiveness of the displayed content by thinning the response datato create a subset of response data that are unconfounded by spatial andtemporal carryover effects; and analyzing the subset of response datausing statistical computing rules.
 10. The computer-implemented methodof claim 8, wherein response data is collected during the second half ofthe response duration.
 11. A computing system comprising: a plurality ofcontent delivery channels, each content delivery channel comprising atleast one display; a computer data store having temporal reach data anda spatial reach data for the content delivery channel; a processorcommunicatively coupled to the data store and configured to executeinstructions that: define at least one hierarchical temporal-spatialunit (HTSU) based on the temporal reach data and the spatial reach data,wherein the at least one HTSU has a response area based on the spatialreach data; derive the response area of the at least one HTSU byreceiving a temporal reach factor radius; modify the temporal reachfactor radius to find a response area radius of the at least one HTSU;receive confound data for at least some of a plurality of differentpieces of content, wherein confound data includes data about whichpieces of content are being compared against one another; and assign theplurality of different pieces of content to the plurality of contentdelivery channels, wherein two pieces of content being compared againstone another are not assigned to one of the plurality of HTSUs.
 12. Thecomputing system of claim 1, further comprising a plurality of HTSU's.13. The computing system of claim 12, wherein the temporal reach factoris the same among at least two of the plurality of HTSU's.
 14. Thecomputing system of claim 12, wherein the temporal reach factor differsbetween at least two of the plurality of HTSU's.
 15. The computingsystem of claim 12, wherein the response area is the same among at leasttwo of the plurality of HTSU's.
 16. The computing system of claim 12,wherein the response area differs between at least two of the pluralityof HTSU's.
 17. The computing system of claim 11, wherein the processoris further configured to execute instructions that receive behavior datawithin the response area of the HTSU during at least a portion of theresponse duration, wherein the collecting of behavior data is associatedwith a location other than the location associated with the presentingof the displayed content.
 18. The computing system of claim 11, whereinthe processor is further configured to execute instructions that: causethe selected content to be displayed on the channel for content deliveryconsistent with the content assignment; and receive response dataindicative of the effects of the displayed content.
 19. The computingsystem of claim 18, wherein the processor is further configured toexecute instructions that: determine the effectiveness of the displayedcontent by thinning the response data to create a subset of responsedata that are unconfounded by spatial and temporal carryover effects;and analyze the subset of response data using statistical computingrules.
 20. The computing system of claim 18, wherein response data iscollected during the second half of the response duration.